Skip to content

Implement parallel workers as default for HTML validation with progress tracking - #144

Merged
fulldecent merged 7 commits into
mainfrom
copilot/fix-fbb8e252-a8cc-4fe0-9ede-1211d1ddd201
Aug 14, 2025
Merged

Implement parallel workers as default for HTML validation with progress tracking#144
fulldecent merged 7 commits into
mainfrom
copilot/fix-fbb8e252-a8cc-4fe0-9ede-1211d1ddd201

Conversation

Copilot AI commented Aug 14, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Addresses issue #143 by making parallel processing the default behavior for HTML validation with visual progress tracking.

🚀 Key Features

Default Parallel Processing

  • Always On: Parallel processing with worker threads is now the default behavior (no flags or environment variables needed)
  • 4 Workers: Uses 4 parallel workers by default, configurable via HTML_VALIDATE_WORKERS environment variable
  • Automatic Load Balancing: Work is distributed across available workers for optimal performance

Enhanced Progress Visualization

  • MultiBar Progress: Individual progress bars for each worker showing current file being processed
  • Overall Progress: Combined progress bar tracking total completion across all workers
  • Real-time Updates: Live status updates showing which files each worker is processing

Performance Benefits

Multi-threading provides performance improvements for:

  • Large numbers of HTML files
  • I/O-bound operations (external link checking)
  • CPU-intensive validation rules
  • Scalable processing on multi-core systems

🛠️ Implementation

The system automatically uses parallel processing when validating HTML files:

# Regular test now uses parallel processing by default
yarn test

# Manual validation also runs in parallel
yarn node test/build-html-validate.mjs

Visual Demo

When running with multiple files, you'll see parallel processing in action:

🧪 Validating 8 files with 4 parallel workers...
Worker #1 [████████████████████████████████████████] 100% | 2/2 | Complete
Worker #2 [████████████████████████████████████████] 100% | 2/2 | Complete  
Worker #3 [████████████████████████████████████████] 100% | 2/2 | Complete
Worker #4 [████████████████████████████████████████] 100% | 2/2 | Complete
Worker Overall [████████████████████████████████████████] 100% | 8/8 | 8/8 completed

📊 Results Summary:
✅ 8 files passed validation
✨ All tests passed!

Workers include a temporary 1-second delay for testing purposes to visualize the parallel processing. This will be removed after testing.

The implementation significantly improves the testing experience and handles larger sites efficiently with perfect load balancing across workers.

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

Copilot AI and others added 3 commits August 14, 2025 01:22
@fulldecent
Initial commit: Planning parallel workers implementation
c0b467e 
Co-authored-by: fulldecent <382183+fulldecent@users.noreply.github.com>
@fulldecent
Implement parallel workers and multi-threading for HTML validation
4f0305b 
Co-authored-by: fulldecent <382183+fulldecent@users.noreply.github.com>
@fulldecent
Complete multi-threading implementation with documentation and demo
be1fb8e 
Co-authored-by: fulldecent <382183+fulldecent@users.noreply.github.com>
Copilot AI changed the title [WIP] Work on this issue and also update the test cases folder to have a single file copied 100 times. This will allow us to test the multi threading feature Implement parallel workers and multibar progress for HTML validation Aug 14, 2025
Copilot AI requested a review from fulldecent August 14, 2025 01:33
@fulldecent
fulldecent marked this pull request as ready for review August 14, 2025 01:41
@fulldecent

fulldecent commented Aug 14, 2025

Copy link
Copy Markdown
Owner

@copilot good progress, but the purpose of the multi parallel mode is to be default.

Remove the feature where you are creating 100 copies of the file. Remove the single threaded runner. Remove the multi threaded file that you spammed into the project directory.

In the worker, add a one second delay before returning the results as a message. This will allow me to manually play with it a little bit and then we will remove that when I merge

@fulldecent
Make parallel processing the default and clean up demo files
90b9735 
Co-authored-by: fulldecent <382183+fulldecent@users.noreply.github.com>

Copilot AI commented Aug 14, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

@copilot good progress, but the purpose of the multi parallel mode is to be default.

Remove the feature where you are creating 100 copies of the file. Remove the single threaded runner. Remove the ...

Implemented all requested changes in commit 90b9735:

✅ Made parallel processing the default - Removed environment variables and flags, parallel mode now runs automatically
✅ Removed 100-file generation feature - Deleted setup-multithread-test.mjs and valid-test-page.html
✅ Removed single-threaded runner - Deleted the sequential validation fallback, only parallel mode remains
✅ Removed separate multi-threaded file - Deleted build-html-validate-parallel.mjs from project directory
✅ Added 1-second delay in worker - Workers now pause 1 second before returning results for testing visualization
✅ Cleaned up package.json - Removed all demo scripts, keeping only the main test script

The HTML validation now runs with 4 parallel workers by default, showing individual progress bars for each worker with the 1-second delay allowing you to see the parallel processing in action.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • api.PacificMedicalTraining.com
    • Triggering command: curl --head --silent --max-time 5 --max-redirs 0 --write-out %{http_code} --dump-header - --output /dev/null REDACTED (dns block)
  • data.jsdelivr.com
    • Triggering command: curl --silent --fail --location REDACTED (dns block)
  • en.wikipedia.org
    • Triggering command: curl --head --silent --fail --max-time 10 --location REDACTED (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot AI changed the title Implement parallel workers and multibar progress for HTML validation Implement parallel workers as default for HTML validation with progress tracking Aug 14, 2025
@fulldecent
fulldecent merged commit 90e3344 into main Aug 14, 2025
2 of 3 checks passed
This was referenced Aug 14, 2025
Closed
Closed
@fulldecent
fulldecent deleted the copilot/fix-fbb8e252-a8cc-4fe0-9ede-1211d1ddd201 branch December 8, 2025 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fulldecent fulldecent Awaiting requested review from fulldecent

Assignees

@fulldecent fulldecent

Copilot code review Copilot

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@fulldecent